A new class of ranking functions for DCG-like evaluation metrics using conditional probability models

In the context of learning to rank for information retrieval [15], we study a general class of “DCG-like” ranking loss functions which include DCG [13] and approximate ERR [6] as specific cases. We then study the Bayes optimal ranking function for this class, which is a function of the conditional distribution of graded document relevance levels. Our main contribution is a novel class of ranking functions building upon the Bayes optimality result. Specifically, our ranking function class uses the conditional expectation of a function of the document features with respect to an ostensible multiclass logistic regression model. We empirically evaluate our proposals with respect to NDCG on seven standard learning to rank datasets [16] and three different surrogate evaluation metrics. Results show our proposed ranking functions improve accuracy across datasets and surrogate evaluation metrics, achieving 10.6% improvement on average and as high as 25% in the best case.